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Dear  Susan, 

This  letter  is  the  sixth  quarterly  progress  report  for  grant  N00014-91-J-1532,  entitled  “On-Line 
Assessment  of  Expertise.”  It  covers  the  period  January  1,  1993  to  March  31,  1993. 

The  main  objective  in  this  report  period  was  to  deal  with  the  combinatorial  explosion  in  the  size 
of  the  belief  networks  needed  to  analyze  regular  problem  solving  behavior.  As  mentioned  in  our 
proceeding  quarterly  progress  reports,  Olae’s  belief  networks  can  get  quite  large  and  intractable 
with  non-toy  knowledge  bases.  We  have  three  approaches  for  handling  this  problem: 

•  We  improved  our  algorithm  that  builds  a  belief  net  incrementally  so  that  it  is  only  as  large 
as  it  needs  to  be  for  the  particular  actions  we  observe  the  human  to  make.  Unlike  earlier 
attempts,  this  algorithm  assigns  the  same  probabilities  to  nodes  as  does  the  intractible  but 
mathematically  correct  approach. 

•  We  tested  a  well-known  Monte  Carlo  technique  (stochastic  simulation,  Pearl,  1986)  for  esti¬ 
mating  probabilities  on  the  nodes  in  very  large  belief  networks.  To  this  we  have  added  three 
enhancements  that  improve  the  estimates.  For  simulated  physics  problem  solving  data,  the 
combined  technique  rapidly  converges  on  the  correct  probabilities. 

•  We  have  developed  and  tested  a  technique  for  simplifying  complex  Bayesian  net  structures. 
As  the  student  solves  several  problems,  tie  Bayesian  model  of  what  they  know  becomes 
ever  more  complex.  At  some  point,  this  model  will  be  too  large  to  use.  Our  simplification 
technique  uses  statistical  clustering  (based  on  Martin  &  Billman,  1994)  to  reduce  both  the 
number  of  nodes  and  the  number  of  links  in  the  model. 

Olae  provides  all  three  of  these  approaches  as  options  for  the  assessor.  On  the  one  hand,  the 
assessor  who  does  not  mind  waiting  can  receive  guaranteed  assessments.  However,  if  assessment 
must  be  rapid  or  must  follow  the  student  through  hundreds  of  problems,  the  assessor  will  want  to 
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choose  some  subset  of  the  heuristic  aids. 


Best  regards, 

Kurt  VanLehn  / 

Assoc.  Prof,  of  Computer  Science 
LRDC  Senior  Scientist 
& 

Joel  Martin 

LRDC  Research  Scientist 

cc:  ONR  Resident  Representative  N66005,  OSU  Research  Center 
Director,  Naval  Research  Laboratory,  Code  2627 
Defense  Technical  Information  Center 
University  of  Pittsburgh  Contracts  Office 
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